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ABSTRACT: 

To provide a disk subsystem that assures the sequence and the coherence of data update 
regarding two or more disk subsystems, and to provide an asynchronous type remote copy 
function, a main center (9) that is a computer system having the configuration of slave 
subsystems (3-2 to 3-n) connected with a master disk subsystem (3-1) secures the coherence 
between data of the main center (9) and the remote center (10) at the temporary suspension by 
repeating temporary suspension and release of the temporary suspension of the remote copy by 
a master subsystem (3-1 ) at predetermined opportunities, and by repeating temporary 
suspension and release of the temporary suspension of the remote copy by slave subsystems 
(3-2 to 3-n) interlocking with it. 
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Subsystems and Total System Therefor 
CLAIM(S) 

1) A total system comprising: a master disk subsystem having a 
control means for sending/receiving data to/from a higher device 
and a means of transferring the data and stopping the data transfer 
to a subsystem located at a remote site at preset intervals; multiple 
slave disk subsystems, each having a control means for 
sending/receiving data to/from a higher device, a memory means 
for storing said data, transmission means connected to said master 
disk subsystem, an inquiring means for asking whether said master 
disk subsystem is in status of transferring data or temporarily 
stopping the data transfer, and a data transfer means of transferring 
said data, based on the result of said inquiry, to the disk subsystem 
at a remote site when the master subsystem is in status of 
transferring the data; multiple remote disk subsystems located at a 
remote distance from said master disk subsystem and slave disk 
subsystems and having a control means for receiving the data 
transferred from said each subsystem and a memory means for 
storing said data; said total system being characterized in that said 



master disk subsystem and slave disk subsystems transfer data to 
said remote subsystems at the preset intervals; thereby securing 
consistency of the data in said master and slave subsystems and of 
the data in said remote disk subsystems during the stopping period 
of data transfer. 

2) A method to remote copy data to a slave subsystem, as cited in 
Claim 1, comprising a step of receiving data from the higher 
device, a step of storing said data in a memory means, a step of 
asking the master subsystem whether the master subsystem is in 
status of transferring data to the remote center, a step of 
transferring said data to the remote center in the case when said 
master subsystem is in status of transferring said data, a step of 
holding the data regarding the position of said stored data in the 
case when said master subsystem is not in status of transferring 
said data, and a step of transferring said data after said master 
subsystem restarts to transfer said data. 

3) A subsystem comprising a controller for sending/receiving data 
to/from the higher device, a memory for storing said data, and a 
data transfer means of transferring the data while temporarily 
stopping the transfer at a time when said data from said higher 
device is preset in other subsystem. 

4) A subsystem, as cited in Claim 3, wherein said subsystem 
periodically performs a temporary stop and release of data transfer. 

5) A subsystem, as cited in Claim 4, which holds, at a time of 
temporary stop of data transfer, the data regarding the stored 
position of the updated data to be transferred in its own subsystem, 
until the temporary stop is released. 
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6) A total system comprising: a first subsystem having a controller 
for sending/receiving data to/from the higher device, a memory 
device for storing said data, and a transfer section for transferring 
said data from said higher device to the subsystem located at 
another site; multiple subsystems, each having a controller for 
sending/receiving data to/from the higher device, a memory device 
for storing said data, a transfer section for transferring said data to 
the subsystem located at another site, and a transmission path 
connected to said master subsystem. 

7) A total system, as cited in Claim 6, wherein said multiple 
subsystems acquire the transfer status of said first subsystem by 
using said transmission path. 

8) A total system having a controller for sending/receiving data 
to/from the higher device, a memory device for storing said data, 
and multiple disk subsystems for transferring said data to the disk 
subsystems located at another site, characterized in that other disk 
subsystems send an inquiry to the first disk subsystem out of 
multiple disk subsystems connected to said higher device as to 
whether they can transfer said data to said disk subsystems at 
another site. 

9) A total system, as cited in Claim 8, wherein whether said data can 
be transferred or not is determined by whether said first subsystem 
is temporarily stopping the data transfer or not. 



10) A total system, as cited in Claim 9, wherein said first disk 
subsystem periodically performs said temporary stop and release of 
data transfer. 

11) A total system, as cited in Claim 9, wherein the data transfer 
between said disk subsystem and said disk subsystem at said 
another site is asynchronously performed. 

1 2) A total system having a first disk subsystem group connected to 
the higher device and a second disk subsystem group for receiving 
the data transfer from said first disk subsystem group, 
characterized in that one of the disk subsystems in the first disk 
subsystem group is a master subsystem for determining whether 
data can be transferred or not when said first disk subsystem group 
tries to transfer data to the second disk subsystem group. 

1 3) A total system, as cited in Claim 12, wherein whether data can 
be transferred or not is determined by whether said master 
subsystem is temporarily stopping the data transfer to said second 
disk subsystem group. 

14) A total system, as cited in Claim 12, wherein said first disk 
subsystem group and said second disk subsystem group are 
connected by a storage area network (SAN). 

1 5) A total system, as cited in Claim 1 2, wherein data transfer 
between said first disk subsystem group and said second disk 
subsystem group is asynchronously performed. 

16) A total system, as cited in Claim 15, wherein the sequence of 
data updating is secured by designating a dedicated volume at a 
time of said asynchronous data transfer. 
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. 17) A method to secure the consistency with the master subsystem 
by a step of receiving data from the higher device, a step of storing 
said data in the memory, and a step of sending an inquiry to the 
master subsystem as to whether or not the master subsystem is in 
status of transferring data to the remote center. 

DETAILED DESCRIPTION OF THE INVENTION 

(0001) 

The present invention pertains to external memory devices for storing 
data and the total system therefore in a computer system, particularly to a 
remote data-copying technology, whereby a group of multiple external 
memory devices on a remote site (subsystems) are connected to another 
group of multiple external memory devices, and duplicate data are created 
by the external memory devices on the remote site (subsystems) without 
using a host computer, which is a higher device. The subsystem mentioned 
here refers to a memory device having a controller for sending and receiving 
data to and from the higher device and a built-in disk device for storing data 
(When the memory device is a disk device, the subsystem is called a disk 
subsystem.). The total system in this context is defined as a system 
consisting of multiple subsystems. 
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(0002) 
(Prior Art) 

To prepare a back-up copy of data against the cases of natural 
catastrophe such as earthquake, a primary center and a remoter center need 
to be apart by a distance of one hundred km - a few hundreds km. 
Accordingly, a few types of external memory systems using a remote 
copying function, wherein duplicate copies are created and stored in the 
subsystems installed on the primary center and the remote center, 
respectively, have been developed and put to practical use; 
(0003) 

A remote copying function is primarily divided into two types: a 
synchronous type and an asynchronous type. 
(0004) 

With the synchronous type, when a command to update data is given 
from the host computer (higher device) to the subsystem and the command 
is about a remote copying function, completion of the data updating is . 
reported to the higher device of the primary center after the updating task 
commanded to the subsystem in the remote center is completed. In such a 
case, the physical distance between the primary center and the remote center 
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has an impact on the performance of the data transmission path, generating a 

time delay (transmission time). 

(0005) 

By contrast, with the asynchronous type, when a command to update 
data is given from the primary center to the subsystem, completion of the 
updating task is reported to the higher device as soon as the updating task is 
completed in the subsystem of the primary center even if object of the 
command is the remote copying function, and in the subsystem in the remote 
center, the updating of data (copying of data) is executed asynchronously 
with the process in the primary center. Since the data updating is completed 
within the processing time used in the primary center, the transmission time 
for storing the data in the remote center is not needed [sic]. 
(0006) 

With the asynchronous type, the content of the subsystem in the 
remote center is not always consistent with that of the primary center. 
Therefore, if the primary center loses the function due to natural catastrophe, 
the data that have not been copied to the remote center may be lost. 
However, the subsystem on the side of the primary center can be accessed at 
the same level of performance when the remote copying function was not 
used. 
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[0007] 

The aforementioned prior art system, however, comes with the 
following problems. 
(0008) 

"Preserving Data Consistency" 

In the process of remote copying, the subsystem of the primary center 
and the subsystem of the remote center are connected with an independent 
communication link. More specifically, when the remote copying is 
performed between multiple subsystems of the primary center and multiple 
subsystems of the remote center, multiple sets of subsystems, each set being 
connected with an independent communication link, are used. 
(0009) 

In this structure, copying data to the remote center is performed by 
each communication link, so the time when the data is updated varies 
depending upon the subsystems. Accordingly, when the primary center 
loses the function, each subsystem in the remote center has stored the data 
up to a different point, and therefore the data in the primary center and that 
in the remote center cannot be consistent once the time cannot be stamped to 
every data. 
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(0010) 

In the case when the remote center thus creates the back-up copy of 
data of the primary center having multiple subsystems, there is a problem in 
securing the sequence of data among multiple subsystems (preserve a 
sequential order of updating). The "sequence" in this context does not 
simply mean the "sequential order of data" but means the consistency of 
updated data in each subsystem. 
(0011) 

It is inevitable that the copying of the updated data to the remote 
center is processed at a time point later than the actual updating process at 
the primary center in asynchronous remote copying. But, the sequence of 
updating must be consistent with that of the primary center. Even in the 
intermittent updating, the sequence of updating executed in the primary 
center must agree with the sequence of updating executed in the remote 
center in consistent state. 
(0012) 

A database generally consists of database body, various logging data, 
and of control data, each being related to one another. When data is updated, 
not only the database body itself but these logging data and 
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control data are also updated to preserve the consistency of the systems. 
Accordingly, if the sequence of updating data is disrupted, the consistent 
state of these data related to the sequence of updating data is also disrupted. 
In the worst case, the entire database can be possibly destroyed. 
(0013) 

"Insertion of Higher Device" 

When the asynchronous remote copying is performed in general 
environment where there are multiple subsystems in the primary center and 
in the remote center and when the higher device commands the subsystems 
to update data, the data regarding the sequence of updating data, such as data 
indicating stamped times, are provided to the data body, and based on said 
data, copying of updated data to the subsystem is generally executed on the 
auxiliary side. For example, like a remote copying process disclosed in 
Japanese Unexamined Published Application 06-2901 25 (U. S. Patent 
5446871), the remote copying function is embodied by inserting the higher 
device. 
(0014) 

According to the technology disclosed in Japanese Unexamined 
Patent Application 06-290125 (U. S. Patent 5446871), the operating system 
and the disk subsystem of the higher device in the primary center and the 
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data mover software and disk subsystem of the higher device in the remote 
center cooperate to generate and send the data regarding the sequence of 
updating data and copy the updated data by using this data. 
(0015) 

(Problems of the Prior Art to Be Addressed) 

It is indeed possible to use a remote copying function while securing 
the sequence of updating data between the primary center and the remote 
center according to the prior art technology. With this technology, however, 
the software program of the higher device and the subsystem both have to 
have a mechanism for using said function and both have to cooperate with 
each other. Therefore, it becomes necessary to install a new dedicated 
software program. Then, it becomes necessary for the user to reconsider the 
system design because of a need of introduction, installation, and inspection 
of the software program and of an increase of load on the CPU. Therefore, it 
takes long time and high cost to introduce this function. 
(0016) 

In addition, in the case when asynchronous remote copying is 
executed only by using the function of the subsystem, consistency in 
sequence of updating data must be preserved only by using the function of 
the subsystem. However, in the case when the data which need to be 
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consistent in sequence of updating are distributed to and stored in multiple 
subsystems, there is a problem that the multiple subsystems do not have a 
means of preserving the sequence of updated data. There also is another 
problem that it is necessary to prevent an increase of overhead caused by 
processing inside the subsystem in order to preserve the sequence of data- 
updating. 
(0017) 

The objective of the present invention is to present an asynchronous 
remote copying function that can secure the consistent state of data only by 
using the function of the subsystem without installing a new software 
program on the higher device and that can be easily introduced without 
reducing the performance of the primary center. 
(0018) 

(Means to Solve the Problem) 

The subsystems of the primary center and of the remote center that are 
located far apart by a distance are connected to each other. In the case when 
there are multiple subsystems, one of the subsystems (hereinafter called a 
master subsystem) is connected to the rest of the subsystems (hereinafter 
called slave subsystems). 



(0019) 

The master subsystem of the primary center temporarily stops the 
remote copying process or releases the temporary stop. These operations 
can be executed at a proper point in time during the remote copying process 
by the use's presetting it in the master subsystem. The subsystem of the 
primary center, upon receiving the updated data from the higher device, 
begins to store the updated data in its own subsystem. 
(0020) 

The master subsystem checks whether or not its own subsystem has 
temporarily stopped a remote copying process. If it has not temporarily 
stopped the remote copying process, the master subsystem sends the updated 
data to the remote center. The slave subsystem, upon receiving the updated 
data from the host, checks the status of the master subsystem by a control bit 
to be mentioned later, and sends the updated data to the remote center only if 
it has confirmed that the master subsystem has not temporarily stopped the 
remote copying process. 
(0021) 

If the slave subsystem finds that the master subsystem has temporarily 
stopped the remote copying process, the slave subsystem will likewise 
temporarily stop the remote copying. When the remote copying process is 
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temporarily stopped, neither master subsystem nor slave subsystem send 
data to the remote center, but the primary center alone performs the data- 
updating process and its subsystems store the updated data. Once the master 
subsystem releases the temporary stop of the remote coping process, the 
master subsystem and the slave subsystems restart to send the updated data 
to the remote center. In this case, the updated data of the primary center 
during said temporary stop is also sent to the remote center. 
(0022) 

Thus, the data are duplicated by asynchronous remote copying 
between both volumes in both subsystems of the primary center and of 
remote center. 
(0023) 

(Embodiment) 

The present invention is explained below with reference to one 
example in which the present invention is applied to a computer system. 
(0024) 

Fig. 1 shows said one example wherein data is duplicated by two 
centers properly selected out of multiple data centers each equipped with a 
computer system. 
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(0025) 

One or more disk subsystems 3 of the primary center 9 and one or 
more disk subsystems 7 of the remote center 10 are connected without the 
higher devices, (host computers) 1 and 8, and both centers constitute a 
remote copying system by performing a task of data duplication. For 
connecting the disk subsystems without using a higher device, for example, 
a storage area network (SAN) can be used. Fig. 4 shows one example of the 
disk subsystem 3 of the primary center 9. 
(0026) 

In the primary center 9 of Fig. 1, the higher device 1 having a CPU for 
conducting data processing is connected to disk subsystem 3 - 1 (master 
subsystem) and to slave subsystems, 3 - 2, 3 - n, via interface cable 2, 
which is a transmission path. 
(0027) 

The disk subsystems, 3-1,3-2, . .., 3 - n, comprises: interface 
controller 1 1 for transmitting/receiving data to/from the higher device 1 ; 
buffer 12, which the higher device 1 refers to and stores the data for updated 
data in; remote copying control data storage 16 for storing the data 
regarding the stored position of the data updated while the remote copying 
function is temporarily stopping; magnetic disk drive 13 as the recording 
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medium for recording said data; micro processor 14 for controiiing the 
exchange of these data; and disk array subsystem controller 17 for 
controlling each of these elements. 
(0028) . 

In addition, the master disk subsystem 3 - 1 is equipped with service 
processor panel 15 on which the user sets up how remote copying is 
performed. The remote copying control data storage 16 stores the control bit 
indicating the present remote copying status based on the control data preset 
on the service processor panel 14, as well as the data regarding the stored 
position of the data updated while the remote copying is temporarily 
stopping. 
(0029) 

The master disk subsystem 3 - 1 is, via the interface cable 4 - 1 , 
connected to the disk subsystem 7 - 1 of remote center 10. The slave disk 
subsystem 3 - 2 is, likewise, connected, via interface cable 4 - 2, to the disk 
subsystem 7-2 of the remote center. The slave disk subsystem 3 - n is 
connected, via the interface cables 4 - n, to the disk subsystem 7 - n of the 
remote center. 
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(0030) 

It is also possible to connect the interface cables, 4-1,4-2, 4 - 
n, to general communication lines by using circuit connection devices. In 
this context, this option is included in the definition of interface cables 4 - 1 , 
...,4-n. 
(0031) 

In the case when there are multiple disk subsystems 3, the disk 
subsystem 3 - 1 is connected, via the interface cable 5, to the disk 
subsystems, 3 -2, 3 - n, other than the disk subsystem 3 - 1, in which are 
stored the data to be remote copied in the primary center. Thus, on the side 
of the primary center 9, the disk subsystem 3, in which the data to be remote 
copied, is connected to one master disk subsystem 3-1 and to the slave disk 
subsystems, 3 - 2, . . ., 3 - n, via the interface cable 5. 
(0032) 

Once the higher device 1 requests the master disk subsystem 3 - 1 to 
input data, the master disk subsystem 3 -1, synchronously with this request, 
inputs said data in the data buffer 12 in its own subsystem, and it also 
commands, asynchronously with its inputting the data in its own buffer 12, 
the disk subsystem 7 - 1 to input the data. Said data that is input in the data 



buffer 12 in its own subsystem is synchronously or asynchronously recorded 

on the magnetic disk drive 13. 

(0033) 

In the method of asynchronous remote copying by inputting the data 
in the remote center, the disk subsystem 3 of the primary center 9 sends the 
updated data to the disk subsystem 7 of the remote center 10 connected to 
the its own subsystem, in the sequential order in which the volume in its own 
subsystem has been updated, and the disk subsystem 7 of remote center 10, 
according to the sequential order of receiving them, sends a set of mode, in 
which to copy the updated data on the volume in its own subsystem, and of 
data to be sent at a point in time properly scheduled by disk subsystem 3, 
without following the sequential order of the volumes updated by the 
subsystem of the primary center 9. The disk subsystem 7 has a mode for 
reflecting the updated data on the volume in its own subsystem without 
following the sequential order of receiving them. 
(0034) 

When the higher disk 1 requests the disk subsystems, 3 - 2, . . ., 3 - n, 
to input the data, the slave disk subsystems, 3 - 2, . . ., 3 - n, synchronously 
with this request, input said data in the data buffer 12 in their own 
subsystems and also look up the status of the remote copying control data 
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storage 16 of the master disk subsystem 3 - 1 . Then, according to the 
remote coping status, they command, asynchronously with said inputting the 
data in the data buffer 12 in their own subsystems, the disk subsystems, 7 - 
2, ... 7 - n, in the remote site to input the data, or determines whether or not 
the data regarding the stored position of the updated data should be stored in 
the remote copy control data storage 16 in their own subsystems. The disk 
subsystems, 7- 1, 7 - 2, . . ., 7 - n, store in the data buffer 12 in their own 
subsystems the data received from disk subsystems, 3 - 1, 3 - 2, . . ., 3 - n, 
connected by interface cable 4 in the data buffer 12. 
(0035) 

More specifically, when the higher device commands the disk 
subsystems, 3 - 1 , 3 - 2, . . ., 3 - n, to input the data, the same data is stored 
in one or multiple disk subsystems, 7-1,7-2, . . . , 7 - n, in the remote 
center 10 depending upon the status of the disk subsystem 3-1. The arrow 
in Fig. 1 shows a flow of data which are to be input by the command of the 
higher device 1. 
(0036) 

Since the master disk subsystem 3 - 1 has the control bit indicating 
the remote copying status in the remote copying control data storage 1 6, it 
can temporarily stop the remote copying process by changing this control bit 



19 



data by the command from the system operator at a proper point in time 
preset by the system operator. When the remote copying is temporarily 
stopped, the disk subsystems, 3 - 1, 3 - 2, . . ., 3 - n, store the updated data in 
the data buffer in their own disk subsystems while simultaneously storing in 
the remote copying control data storage 16 the data regarding the stored 
position of the updated data that is commanded to be input, after the 
temporary stop, and they do not command the disk subsystems, 7-1,7-2, 
7 - n, to input the updated data. 
(0037) 

In the system of the present invention, since the remote copying is 
temporarily stopped, the data existing in the primary center 9 when the 
remote copying is temporarily stopped exist in all the subsystems in the 
remote center 10. In other words, at a point in time when the temporary stop 
of remote copying occurred, the data in the primary center 9 and the data in 
the remote center 10 are consistent. Therefore, stamping of a time for 
securing the consistency needs not be done to the data, and the remote 
copying can be implemented without the higher device even for an open 
system in which the time data is not provided by the higher device. 
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(0038) 

The disk subsystems, 3 - 1 , 3 - 2, .. ., 3 - n, can release said 
temporary stop of remote copying based on the command of the system 
operator at a proper point in time preset at master disk subsystem 3 -1 by the 
system operator. 
(0039) 

Once the temporary stop is released, the disk subsystems, 3-1,3-2, 
. . ., 3 - n, command the disk subsystems, 7 - 1 , 7- 2, . . ., 7 - n, to input the 
data updated during the temporary stopping period and if they receive the 
command from the higher device 1 to input the data, they, synchronously 
with the command, input said data in the data buffer 12 in their subsystems. 
Also, they command, asynchronously with said inputting the data in the data 
buffer 12 in their own subsystems, the disk subsystems, 7- 1 , 7- 2, . . ., 7- n, 
of the remote site to input the data. 
(0040) 

By use of the aforementioned structure, same data can be preserved in 
the volume, which is the object of remote copying, of the disk subsystem 3 
of the primary center and in the volume of the disk subsystem 7 of remote 
center 10 if a time delay in updating can be disregarded. Moreover, while 
the remote copying is temporarily stopped in the master subsystem 3 -1, the 
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data status in each disk subsystem 3 of primary center 9 at a time when the 
master subsystem 3 - 1 has stopped, i.e. the data status having consistency at 
said point in time is preserved in each disk subsystem 7 of the remote center 
10. 

(0041) ' 

The temporary stop and release of remote copying can be set per a 
volume pair unit of remote copying. It is also possible to arrange multiple 
volume pairs into one volume group and change the status by a volume 
group unit. The temporary stop and release of remote copying can be 
displayed on the subsystems 3 and 7, the consoles of the higher devices 1 
and 8, or on the monitor used when these system are controlled. By this, the 
user can recognize if the remote copying is in progress at present and if the 
remote copying is performed per a volume group unit. 
(0042) 

The time duration between the temporary stop and the release of 
remote copying can be selected properly by the user as long as it is so short 
that the temporary stop is released before all the data before the temporary 
stop occurred are copied to the remote center and new data are copied to the 
remote center 10, destroying the consistency of the data between the primary 
center 9 and the remote center 10. 
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(0043) 

The following example takes into account the time for copying and 
storing the data in subsystem 7 in the remote center 10 at a time of 
temporary, stop of remote copying; in this example, remote copying is 
performed from the primary center 9 to the remote center 10 for 30 minutes; 
the remote copying is temporarily stopped for the subsequent 30 minutes; 
subsequently, the temporary stop is released, and the remote copying is 
performed again for the subsequent 30 minutes. If the copying time is not 30 
minutes in the remote center, the duration of temporary stop may be changed 
in accordance with the copying time period or the time duration between the 
temporary stop and the release may be selected regardless of copying time 
period. 
(0044) 

It will be also possible to construct a structure wherein the slave disk 
subsystem needs not consult with the master subsystem on the control bit if 
the master disk subsystem preliminarily provides, via the interface cable 5, 
the slave disk subsystems, 3 - 2, . . ., 3 - n, with the control bit indicating the 
remote copying status stored in the remote copying control data storage 16 
of the master disk subsystem 3 - 1 . In such a case, the status of the slave 
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subsystem is stored in the remote copying control data storage 16 of the 
slave disk subsystem like the case with the remote copying control data 
storage 16 of the master disk subsystem. 
(0045) 

The higher device 8 is connected, via the interface cable 6, to the disk 
subsystems, 7 - 1, 7 - 2, 7 - 1, in the remote center 10, and it is a central 
processing device to which data is referred, and which stores the updated 
data in the disk subsystems, 7 - 1 , 7- 2, . . ., 7 - n. The higher device 8 can 
be a substitute of the higher device 1 when the higher device 1 cannot 
function due to catastrophe or failure. In addition, the higher device 8 can 
execute a process different from that performed by the higher device 1 in the 
primary center 9 independently from the higher device 1 by using the data 
stored in the disk subsystems, 7-1, 7-2,..., 7 - n. 
(0046) 

However, if the higher device 8 needs not perform a process to the 
disk subsystems, 7 - 1, 7 - 2, .. ., 7 - n, and does not have a substituting 
function of higher device 1, the higher device 8 will not be needed. On the 
contrary, by installing the higher device 8 and connecting the disk subsystem 
7 - 1 to the other disk subsystems, 7 - 2, . . ., 7 - n, with interface cable 5, to 
provide it with the same structure as that of master disk subsystem 3 - 1 in 
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the primary center 9, the primary center 9 in Fig. 1 can function as a remote 

center and the remote center 9 as a primary center. 

(0047) 

As one form of embodying the present invention, the data duplication 
method and operation are explained below with reference to Fig. 2. 
(0048) 

The file, volume, disk subsystem 3 in which the data to be duplicated 
are stored are preliminarily selected by the operator according to a need of 
duplication, i.e., remote copying. The operator presets, in the remote 
copying control data storage 16 in the master disk subsystem 3 - 1 by 
referring to the higher device 1 or service processor 15, the relationship of 
the file, volume, and disk subsystem 3 to be selected and of the file, volume, 
and disk subsystem 7 for storing the copied data, as well as a need of 
constantly preserving the consistency in sequence of data updating. 
Generally, only the logging file containing the history of database updating 
is preset to preserve the consistency in sequence of updating. In this 
example, however, the presetting to preserve the consistency in sequence of 
updating data is not done regardless of types of files. 
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(0049) 

With the master subsystem 3 - 1, the time to temporarily stop remote 
copying and the time to release it are preset. This presetting can be 
commanded by the higher device 1 , so the time for the higher device to issue 
a command can be preliminarily scheduled by the program of the higher 
device that supports automation of the operation. 
(0050) 

In the case of using the disk subsystem 3 to which the dedicated 
console or service processor 1 5 can be connected or installed for said 
selection and presetting, the higher device 1 is not used, but the console or 
service processor can perform said selection and presetting. In this example, 
the higher device 1 is not used, but the operator presets so that the periodical 
temporary stops and releases of remote copying can be executed in the 
master subsystem 3-1 by using a time value stored in the disk subsystem. 
(0051) 

Fig. 2 shows the flow of operation wherein said selection and 
presetting are done from the dedicated console. The temporary stop and 
release of remote copying are preset per a volume pair unit to be remote- 
copied (Step 1 : S 1 in the figure.). Generally, all the volume pairs to be 
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remote-copied are defined as one volume group, and all the volumes in one 

volume group are preset to share the same status. 

(0052) 

In this example, all the volumes in the disk subsystem 3 are the 
objects of remote copying. Accordingly, hereinafter, a remote copying status 
is described per a disk subsystem unit, not per a volume or volume group. In 
this example, although the details are not described, in the database and 
logging file, the presetting is done per every volume group, and the 
temporary stop and release of remote copying needs not be preset on the 
volume in which to store the logging file. 
(0053) 

As to the method of presetting the file and volume which are the 
objects of remote copying, there is a method to designate a specific address 
of specific volume or disk system, or a method to select it from a proper 
range of addresses by the control program in the disk subsystem. The 
example shows the initial presetting wherein the pass, pair, and temporary 
stop and release are preset. 
(0054) 

Once the higher device 1 commands the disk subsystem 3 - 1 to input 
data (hereinafter referred to as a write command) (Step 2), the disk 
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subsystem 3 - 1 stores the data in its own disk subsystem based on the write 
command (Step 3), reports the completion of the write command to the 
higher device 1 after having written the data in its own subsystem (Step 4). 
(0055) 

Also, once the higher device issues the write command to the disk 
subsystems, 3 - 2, . . ., 3 - n (Step 2), the disk subsystems, 3 - 2, . . ., 3 - n, 
store the data in their own subsystems based on the write command (Step 5). 
The write command in this context includes the command to write data and 
transfer of data to be written. The user presets on the higher device 1 to 
which subsystem the higher device 1 issues a command (Step 1). 
(0056) 

The disk subsystem 3-1, upon receiving the write command, 
discriminates the remote copying status by referring to the control bid in the 
remote copying control data storage 16 that indicates the remote copying 
status of its own subsystem (Step 6 in Fig. 3). If its own subsystem is 
temporarily stopping the remote copying, the disk subsystem 3 - 1 does not 
send the updated data to the disk subsystem 7 - 1 of remote center 10 
connected to its own subsystem, but stores in its own subsystem the data 
regarding the stored position of the updated data. (Step 7). If its own 
subsystem is not in the status of temporarily stopping remote copying, the 
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disk subsystem 3 - 1 issues the write command to the disk subsystem 7 - 1 at 
a time determined by the processing capability of its own subsystem. (Step 
8) 

(0057) 

If the disk subsystem is holding the data regarding the stored position 
. of the data updated during the temporary stopping period of remote copying, 
it determines the data at said stored position as the object to be sent to the 
disk subsystem 7 - 1 of the remote center 10, issues the write command to 
write said data, and deletes the data regarding the stored position of the 
updated data after having completed the write command process. 
(0058) 

The disk subsystems, 3 - 2, . . ., 3 - n, upon receiving the write 
command, send, via the interface cable 5, to disk subsystem 3 - 1 an inquiry 
about the status of the disk subsystem 3-1 and acquire/look up the control 
bit indicating the remote copying status of the disk subsystem 3-1 (Step 9) 
to check out whether the disk subsystem 3 - 1 is in the status of temporarily 
stopping the remote copying (Step 10). 
(0059) 

If the disk subsystem 3 - 1 is in the status of temporarily stopping the 
remote copying, the disk subsystems, 3 - 2, . .., 3 - n, will store in their own 
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subsystems the data regarding the stored position of the updated data (Step 
12), and report the completion of the write command process to the higher 
device 1 (Step 13). 
(0060) 

If the disk subsystem 3 - 1 is not in the status of temporarily stopping 
the remote copying, the disk subsystems, 3 - 2, . . ., 3 - n, report the 
completion of write command to the higher device 1 (Step 14), and issue the 
write command to the disk subsystems, 7 - 2, . . ., 7 - n, at a time 
determined by the processing capability of their own subsystems. In 
addition, if the disk subsystems, 3 - 1, . . ., 3 - n, are holding the data 
regarding the stored position of the updated data, they determine said 
position data as the object to be sent to the disk subsystems, 7 - 2, 7 - n, 
of the remote center 9 (Step 15), and delete the position data of the updated 
data after having completed the write command. 
(0061) 

More specifically, when the disk subsystem 3 - 1 is in the state of 
temporarily stopping the remote copying, the other subsystems of the 
primary center 9 that are connected to the disk subsystem 3 - 1 enter the 
state of temporarily stopping the remote copying by the write command of 
the higher device 1. When the disk subsystem 3 - 1 is not in the status of 
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temporarily stopping the remote copying, it remote copies the data to the 
other disk subsystems of primary center 9 that are connected to the disk 
subsystem 3 - 1 by the write command from the higher device 1 . 
(0062) 

The presetting may also be done as follows: in the case when the 
remote copying status of the disk subsystem 3 - 1 is changed, the disk 
subsystems, 3 - 1 , . . ., 3 - n, report the change to the disk subsystem 3-1 
(Step 9' not shown in the figure) in stead of sending an inquiry to the disk 
subsystem 3 - l(Step 9), or the disk subsystem 3 - 1 reports the change to 
the disk subsystems, 3 - 2, . . ., 3 - n, in the case when the remote copying 
status of the disk subsystem 3 - 1 is changed, as mentioned earlier. 
(0063) 

With this presetting, the disk subsystems 3 - 2, . . ., 3 - n, need, like 
the subsystem 3 - ldoes, to store the remote copying status. Therefore, if 
the disk subsystem 3 - 1 is in status of temporarily stopping the remote 
copying in step 10, the disk subsystems, 3 - 2, . . ., 3 - n, change the copying 
status to the temporarily stopping the remote copying (Step 1 1 not shown in 
the figure). If the disk subsystem 3 - 1 is not in the state of temporary stop 
of remote copying, the disk subsystems, 3 - 1 , . . ., 3 - n, change the remote 



copying status to the release of temporary stop (Step 1 1' not shown in the 

figure). 

(0064) 

In addition, even if the presetting is done so that the disk subsystems, 
3 - 2, . . ., 3 - n, send the inquiry to the disk subsystem 3 - 1 , they may store 
the remote copying status in therir own subsystems and perform the step 1 1 
and 1 1 ' if the remote copying status in their own disk subsystems needs to 
be displayed. 
(0065) 

The disk subsystems, 7 - 1, . . ., 7 - n, upon receiving the write 
command from the disk subsystems, 3 - 1, . . ., 3 - n, process the write 
command, in other words, store the data in the data buffer 12 in their own 
subsystems (Step 16). 
(0066) 

The disk subsystems, 7 -1 , 7 - 2, . . . , 7 - n, having completed the 
process of write command, in other words, having stored the data in the data 
buffer 12 in their own subsystems, report the completion of the write 
command process to the disk subsystems, 3-1,3-2, ...,3-n (Step 1 7). 
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(0067) 

Once the temporary stop of remote copying is released, the disk 
subsystems, 3 - 1, 3 - 2, . . ., 3 - n, command the disk subsystems, 7-1,7- 
2, . . ., 7 - n, to input the data regarding the stored position of the data 
updated after the remote copying has come to a temporary stop in their own 
subsystems. If the write command is issued from the higher device 1 to the 
disk subsystems, 3 - 1, . . ., 3 - n, to write the same data, the disk 
subsystems, 3 - 1 , . . . , 3 - n, input said data in the data buffer 1 2 in their own 
subsystems synchronously with the above write command, and 
asynchronously with this inputting of said data, they command the disk 
subsystems, 7 - 1 , . . . , 7 - n, in the remote site to input said data. 
(0068) 

According to the present invention, the data input in the higher device 
1 are not only stored in the disk subsystems, 3 - 1 , . . ., 3 - n, but also copied 
to and stored in the disk subsystems, 7 - 1 , . . ., 7 - n. In addition, the data 
status in the disk subsystems, 3 - 1 , . . . , 3 - n, at a time when the disk 
subsystem 3 - 1 has come to a temporary stop of remote copying is 
generated in the disk subsystems, 7 - 1, . . ., 7 - n, of the remote center 10. 
In the case when the primary center 9 falls victim to a catastrophe, the data 
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can be recovered by using the data in the subsystems, 7 - 1 , . . ., 7 - n, so the 

operation can be restarted. 

(0069) 

On the side of remote center 10, the data in the disk subsystems, 7 - 1 , 
. . ., 7 - n, are duplicated and stored by using the disk subsystem volume 
copying function while the remote copying is temporarily stopped. By thus 
creating the copies, the operation can be restarted by using the stored copies 
even if the primary center 9 has fallen victim to a catastrophe, therefore, has 
lost the consistency of data during the operation of remote copying to the 
disk subsystems, 7-1, . . ., 7 - n. 
(0070) 

This operation of the system is executed only by the functions of the 
disk subsystems, eliminating a burden on the processing capability of the 
higher device. 
(0071) 
(Advantage) 

According to the present invention, a remote copying system that can 
be easily installed while securing consistency of updated data to the extent 
expected by the user can be implemented only by changing the functions of 
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the subsystems without introducing a new software application into the 

higher device. 

(0072) 

In addition, when data is exchanged back and forth between the 
primary center and the remote center, the data is processed by a group unit, 
so the data transmission efficiency is improved, while reducing the control 
overhead of the subsystems caused by controlling to preserve the 
consistency in sequence of updating data, by which a high performance 
back-up system against a catastrophe that can be easily introduced can be 
presented. 

BRIEF DESCRIPTON OF THE DRAWINGS 
Fig. 1 shows the whole structure of the remote copying system as one 

embodiment example of the present invention. 

Fig. 2 shows a flowchart of the process in the remote copying system. 
Fig. 3 shows a flowchart of the process in the remote copying system 

of Fig. 2. 

Fig. 4 shows the structure of the master disk subsystem and slave disk 
subsystem in the primary center. 

1. higher device 

2. interface cable 
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3. disk subsystem 

4. interface cable 

5. interface cable 

6. interface cable 

7. disk subsystem 

8. higher device 

9. primary center 

10. remote center 

11. interface controller 

12. data buffer 

13. magnetic disk drive 

14. micro processor 

1 5. service processor panel 

16. remote copying control data storage 

17. disk array subsystem controller 



